cinqict

Thursday, December 10, 2015

Oracle Access Manager (OAM) 11.1.2.3 Header Size Limit

For our client we are using a setup with Oracle OAM 11.1.2.3 behind a 11g OHS server. The client uses Single Sign On based on Kerberos using a Windows 2008 AD/KDC server.

On the test environment, using test accounts, things are working fine. On the prod environment, though, some users are were not able to log in.

The following error was seen in the managed server log:

<Error> <oracle.oam.engine.authn> <BEA-000000> <org.ietf.jgss.GSSException, major code: 11, minor code: 0
major string: General failure, unspecified at GSSAPI level
minor string: Length of input stream read does not match size of the inner context token
org.ietf.jgss.GSSException, major code: 11, minor code: 0
major string: General failure, unspecified at GSSAPI level
minor string: Length of input stream read does not match size of the inner context token
at com.ibm.security.jgss.mech.spnego.SPNEGOContext.acceptSecContext(SPNEGOContext.java:1)
at com.ibm.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:268)
at com.ibm.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:178)
at oracle.security.am.engine.authn.internal.executor.SPNEGOLoginModule$1.run(SPNEGOLoginModule.java:158)

So, we had to figure out what the difference was between the users.

We saw differences between the (spnego) token size, which led to differences between sizes in the headers that were sent from the client to the webtier and, in the end, to the oam server.

There are limits on the OHS that limit the header size. These can be set by the LimitRequestFieldSize in the httpd.conf. But, this would result in a 400 Bad Request Error which was NOT the case. To rule this out, we set this value to 1024 which indeed caused a 400 error. In this case, though, the request did not reach the oam managed server and the error was not seen.

We did see the header entering the oam managed server using extended format logging on the access logs of the MANAGED server. So NOT the access log of the webtier! Using this logging, we could see that the ‘breaking point’ of the error was at exactly 4000 bytes. Sounds like a configurable value, right?

In fact, from OAM 11.1.2.3, there is a new limit on the header size on the OAM Managed server itself!

You can see the limit using wlst from $ORACLE_HOME/common/bin/wlst.sh

Connect to the admin server, and run:

displayTrustedInputConfig()

You can see that the default is set to 4000 (bytes, that is).

Use, the following commands to set the values you want:

configTrustedInputs(name="DEFAULT_HEADER",maxSize="12000")
configTrustedInputs(name="DEFAULT_PARAMETER",maxSize="12000")
configTrustedInputs(name="DEFAULT",maxSize="12000")

And restart the oam managed servers

Now, in this example I use 12000. But, you might want to take into consideration that a SPNEGO ticket has a max size of 12392 bytes. That is only the ticket! So, to rule out the oam managed server as a potential bottleneck, you might want to set this limit to 15000 or so, using some slack for overhead.

Wednesday, November 25, 2015

VirtualBox Installatie Weblogic 12 Cursus

 

Deze blog is specifiek voor de weblogic 12 cursus voor Cinq.

Download VirtualBox voor je specifieke OS van de volgende site:

http://www.oracle.com/technetwork/server-storage/virtualbox/downloads/index.html

Versie 5.0.10 volstaat

Installeer de gedownloade software op je PC en open Oracle VirtualBox

Ga naar File > Preferences > Network > HostOnly Networks en maak een nieuw Network aan door op het groene plusje te klikken

image

Wanneer het network is aangemaakt, dubbelklik je hierop om deze te configureren

Zorg ervoor dat je onderstaande instellingen overneem:

image

image

 

Hierna dien je je pc te herstarten en daarna Oracle VirtualBox weer te starten

Ga nu naar File > Import Appliance en selecteer de image voor de weblogic cursus )ova bestand):

 

image

Klik op Next

Zorg ervoor dat je Reinitialize the MAC Address UIT laat staan !!!

image

Klik op Import

Wanneer de import gedaan is, zie de je virtual host terug in je VirtualBox Manager scherm.

Rechts-klik op de virtualbox en selecteer Settings > Network

image

Zorg ervoor dat Adapter 1 enabled is en attached is met Host-Only Adapter

In Name hoort de HostOnly adapter te staan, welke in de eerdere stap is aangemaakt.

Start vervolgens de VirtualBox op en login met de oraoas user.

Controleer met ifconfig –a of het juiste ipaddress is verkregen. Dit dient 192.168.1.121 te zijn.

 

Zorg er voor dat je een X sessie kunt openen naar de VirtualBox server vanaf je PC.

Voor Linux kun je bijvoorbeeld gebruik maken van ssh –Y

Voor Windows kun je via Putty gebruik maken van Xming

Monday, November 9, 2015

Weblogic Monitoring MBEANS with jconsole

 

With jconsole (comes with the oracle jdk) you can connect to a running local process or to a remote process. The last one is a bit tricky.

First of all, to make use of all the functions, you have to make the wlfullclient.jar and pass this to the classpath when starting jconsole.

To make the wlfullclient.jar, install a local middleware ( I used weblogic 12.1.3 for monitoring weblogic 12 as well as 10). Creation of a domain is not needed.

With the middleware installed, change directories to the WL_HOME/server/lib directory. Then, run the following:

java -jar wljarbuilder.jar

wlfullclient.jar will be created in $WL_HOME/server/lib

Now, export your JAVA_HOME and WL_HOME as system variables and run the following to start jconsole:

jconsole -J-Djava.class.path=$JAVA_HOME/lib/jconsole.jar:$JAVA_HOME/lib/tools.jar:$WL_HOME/server/lib/wlfullclient.jar -J-Djmx.remote.protocol.provider.pkgs=weblogic.management.remote

You can add –debug at the end of you run into problems…

You can add -J-DsocksProxyHost=localhost -J-DsocksProxyPort=1111 if you are using tunneling/proxying to a remote server.

When starting weblogic, the out log file will show the ports that are opened and which protocol is available:

<Nov 9, 2015 12:09:59 PM CET> <Notice> <Server> <BEA-002613> <Channel "Default[3]" is now listening on 0:0:0:0:0:0:0:1:7021 for protocols iiop, t3, ldap, snmp, http.>
<Nov 9, 2015 12:09:59 PM CET> <Notice> <Server> <BEA-002613> <Channel "Default[2]" is now listening on fe80:0:0:0:a00:27ff:fe71:4f97:7021 for protocols iiop, t3, ldap, snmp, http.>
<Nov 9, 2015 12:09:59 PM CET> <Notice> <Server> <BEA-002613> <Channel "Default[1]" is now listening on 10.0.3.15:7021 for protocols iiop, t3, ldap, snmp, http.>
<Nov 9, 2015 12:09:59 PM CET> <Notice> <Server> <BEA-002613> <Channel "Default[4]" is now listening on 127.0.0.1:7021 for protocols iiop, t3, ldap, snmp, http.>
<Nov 9, 2015 12:09:59 PM CET> <Notice> <Server> <BEA-002613> <Channel "Default" is now listening on 127.0.1.1:7021 for protocols iiop, t3, ldap, snmp, http.>
<Nov 9, 2015 12:09:59 PM CET> <Notice> <WebLogicServer> <BEA-000329> <Started the WebLogic Server Administration Server "AdminServer" for domain "wl_domain" running in production mode.>

Use this port number in the remote connect string :

service:jmx:iiop://10.0.3.15:7021/jndi/weblogic.management.mbeanservers.runtime

image

 

Now, go to the MBEANS tab and search for the mbean you wish to monitor.

Double Click on any value integer on the right pane, and a graph will automatically be created for you

image

 

 

NOTE: If you have enabled the Adminstration Channel, the weblogic user will be forced to use a secure connection. In this case, the way to go is to create a separate user with monitor roles and use this in the jconsole credentials.

 

Happy monitoring!

Monday, October 12, 2015

oidprovtool in OID 11.1.1.9 Bug Exception while loading jdbc diver

In OID 11.1.1.9, the oidprovtool does not load the correct jdbc driver, which causes a connect error while creating provisioning profiles.

To fix this issue, prepend the ojdbc6,jar  to the classpath in the $ORACLE_HOME/bin/oidprovtool

Original:

CLASSPATH=${LDAPJCLNT11}:${ORACLE_HOME}/jlib/oraclepki103.jar:$ORACLE_HOME/j2ee/home/jps-api.jar:$ORACLE_HOME/j2ee/home/jps-internal.jar

With added jar:

CLASSPATH=/w001/app/mwhome_idm/Oracle_IDM1/jdbc/lib/ojdbc6.jar:${LDAPJCLNT11}:${ORACLE_HOME}/jlib/oraclepki103.jar:$ORACLE_HOME/j2ee/home/jps-api.jar:$ORACLE_HOME/j2ee/home/jps-internal.jar

Thursday, October 1, 2015

Oracle OAM 11.1.2.3 Error Opening of file based keystore

 

When starting an Oracle OAM 11.1.2.3 Admin Server, I ran into the following error.

Error message: JPS-06514: Opening of file based keystore failed.

(See full error message at the bottom)

This domain had been running for weeks, so it was a spontaneous error, which occurred after the AIX machine running the OAM stack had crashed.

To resolve this issue, I removed the $DOMAIN_HOME/AdminServer direcory and started the AdminServer again. Had to create a new boot.properties file, but at least the problem was solved!

 

<Oct 1, 2015 1:31:47 PM CEST> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-06514: Opening of file based keystore failed.
weblogic.security.SecurityInitializationException: The loading of OPSS java security policy provider failed due to exception, see the exception stack trace or the server log file for root cause. If still see no obvious cause, enable the debug flag -Djava.security.debug=jpspolicy to get more information. Error message: JPS-06514: Opening of file based keystore failed.
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.loadOPSSPolicy(CommonSecurityServiceManagerDelegateImpl.java:1402)
        at weblogic.security.service.CommonSecurityServiceManagerDelegateImpl.initialize(CommonSecurityServiceManagerDelegateImpl.java:1022)
        at weblogic.security.service.SecurityServiceManager.initialize(SecurityServiceManager.java:873)
        at weblogic.security.SecurityService.start(SecurityService.java:141)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsRuntimeException: JPS-06514: Opening of file based keystore failed.
        at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:190)
        at oracle.security.jps.internal.policystore.TenantJavaPolicyProvider.<init>(TenantJavaPolicyProvider.java:161)
        at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:306)
        at oracle.security.jps.internal.policystore.JavaPolicyProvider.<init>(JavaPolicyProvider.java:279)
        at java.lang.J9VMInternals.newInstanceImpl(Native Method)
        Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.JpsException: JPS-06514: Opening of file based keystore failed.
        at oracle.security.jps.internal.policystore.PolicyUtil.getDefaultPDPService(PolicyUtil.java:3180)
        at oracle.security.jps.internal.policystore.PolicyUtil.getPDPService(PolicyUtil.java:3480)
        at oracle.security.jps.internal.policystore.PolicyUtil.getPDPService(PolicyUtil.java:3466)
        at oracle.security.jps.internal.policystore.PolicyDelegationController.<init>(PolicyDelegationController.java:188)
        at oracle.security.jps.internal.policystore.TenantJavaPolicyProvider.<init>(TenantJavaPolicyProvider.java:161)
        Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.service.keystore.KeyStoreServiceException: JPS-06514: Opening of file based keystore failed.
        at oracle.security.jps.internal.keystore.file.FileKeyStoreManager.openKeyStore(FileKeyStoreManager.java:458)
        at oracle.security.jps.internal.keystore.file.FileKeyStoreManager.openKeyStore(FileKeyStoreManager.java:392)
        at oracle.security.jps.internal.keystore.file.FileKeyStoreServiceImpl.doInit(FileKeyStoreServiceImpl.java:128)
        at oracle.security.jps.internal.keystore.file.FileKeyStoreServiceImpl.start(FileKeyStoreServiceImpl.java:901)
        at oracle.security.jps.internal.keystore.FarmKeyStoreServiceImpl.initialize(FarmKeyStoreServiceImpl.java:148)
        Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.service.keystore.KeyStoreServiceException: JPS-06519: Failed to get/set credential with map fks and key null in bootstrap credstore. Reason oracle.security.jps.service.keystore.KeyStoreServiceException: JPS-06519: Failed to get/set credential with map fks and key current.key in bootstrap credstore. Reason null
        at oracle.security.jps.internal.keystore.util.KeyStoreServiceUtil.getMasterKey(KeyStoreServiceUtil.java:453)
        at oracle.security.jps.internal.keystore.file.FileKeyStoreManager$3.run(FileKeyStoreManager.java:412)
        at oracle.security.jps.internal.keystore.file.FileKeyStoreManager$3.run(FileKeyStoreManager.java:410)
        at java.security.AccessController.doPrivileged(AccessController.java:419)
        at oracle.security.jps.internal.keystore.file.FileKeyStoreManager.openKeyStore(FileKeyStoreManager.java:410)
        Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.service.keystore.KeyStoreServiceException: JPS-06519: Failed to get/set credential with map fks and key current.key in bootstrap credstore. Reason null
        at oracle.security.jps.internal.keystore.util.KeyStoreServiceUtil.getCurrentMasterKeyAlias(KeyStoreServiceUtil.java:303)
        at oracle.security.jps.internal.keystore.util.KeyStoreServiceUtil$3.run(KeyStoreServiceUtil.java:446)
        at oracle.security.jps.internal.keystore.util.KeyStoreServiceUtil$3.run(KeyStoreServiceUtil.java:444)
        at java.security.AccessController.doPrivileged(AccessController.java:419)
        at oracle.security.jps.internal.keystore.util.KeyStoreServiceUtil.getMasterKey(KeyStoreServiceUtil.java:444)
        Truncated. see log file for complete stacktrace
Caused By: oracle.security.jps.service.credstore.CredStoreException: JPS-01061: Access to bootstrap credential store denied to application code.
        at oracle.security.jps.internal.credstore.ssp.SspCredentialStore.getCredential(SspCredentialStore.java:643)
        at oracle.security.jps.internal.keystore.util.KeyStoreServiceUtil.getCurrentMasterKeyAlias(KeyStoreServiceUtil.java:292)
        at oracle.security.jps.internal.keystore.util.KeyStoreServiceUtil$3.run(KeyStoreServiceUtil.java:446)
        at oracle.security.jps.internal.keystore.util.KeyStoreServiceUtil$3.run(KeyStoreServiceUtil.java:444)
        at java.security.AccessController.doPrivileged(AccessController.java:419)
        Truncated. see log file for complete stacktrace

Friday, August 14, 2015

Coherence Warning on Oracle Weblogic 12c

 

The error

This was seen on a Oracle Weblogic 12.1.3 server on AIX

PreferredUnicastUdpSocket failed to set receive buffer size to 1444 packets (2MB); actual size is 42%, 609 packets (864KB). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance.>

The fix

To fix this issue, configure your OS to allow for larger buffers.

On Linux execute (as root):

sysctl -w net.core.rmem_max=2096304
sysctl -w net.core.wmem_max=2096304

On AIX execute (as root):

no -op rfc1323=1
no -op sb_max=4194304
no -op udp_recvspace=4194304
no -op udp_sendspace=65536

(Note that AIX only supports specifying receive buffer sizes of 1MB, 4MB, and 8MB)

Wednesday, August 12, 2015

Oracle Weblogic 12.1.3 Infrastructure + Webtier 12.1.3 Install

 

Introduction

In Oracle Fusion Middleware 12c, things have changed, compared to 11g.

One of the differences is the Weblogic Server installation. In 11g you could extend your already existing, plain Weblogic Domain with FMW Products like Webtier 11g. In 12c this has changed a bit. There is the Weblogic Server 12.1 Server which you can use for java EE deployments. But there is also a FusionMiddeware Inftastructure installer, which should be used as a base for other Fusion Middleware components like SOA, OSB, Webtier etc.

You should think about this, when designing a Fusion Middleware environment. When installing the ‘plain’ Weblogic Server, you are not able to extend this with, let’s say, Webtier 12c. When installing, Webtier won’t take the Weblogic12 as a valid Oracle Home directory!

So, with that in mind, we are going to install the Fusion Middleware Infrastructure 12.1 3 and, on top of that, the Oracle Webtier 12.1.3.

Ingredients

The software used in this example:

  • A Oracle jdk 1.7 installed at /w001/app/jdk
  • fmw_12.1.3.0.0_infrastructure.zip (FusionMiddleware Infrastructure, Generic)
  • fmw_12.1.3.0.0_ohs_linux64_Disk1_1of1.zip (Webtier, specific for Linux 64)
  • Oracle Linux 6, 64 bits
  • Oracle Database 12c

Installing the Fusion Middleware Infrastructure 12.1.3

Unzip the fmw_12.1.3.0.0_infrastructure.zip file. This will unpack a file called fmw_12.1.3.0.0_infrastructure_Disk1_1of1.jar

Run the following command to open the installer

$JAVA_HOME/bin/java –jar fmw_12.1.3.0.0_infrastructure_Disk1_1of1.jar

This will open the inventory setup screen, in which you have to provide a valid directory for the oraInventory

Screenshot_8

Next, the Welcome screen appears. Press Next to continue

Screenshot_1

Enter a valid Oracle Home directory. In 11g, this was called a Middleware Home

Screenshot_2

Select Fusion Middleware Infrastructure in the next screen and press Next

Screenshot_3

OS and Java checks are run

Screenshot_4

A summary screen appears. Press Install to begin the installation.

Screenshot_5

If all goes well, press Next if the installation completes

Screenshot_6

And finally press Finish

Screenshot_7

That’s all for the installation. Next step is the installation of the Oracle Webtier.

Installing the Oracle Webtier 12.1.3

We are now going to install the Webtier component onto the Infrastructure

Unzip the fmw_12.1.3.0.0_ohs_linux64_Disk1_1of1.zip file, which will unpack a single file called fmw_12.1.3.0.0_ohs_linux64.bin

This file is the actual installer which can be directly run. Make sure the file has execute rights, though.

./fmw_12.1.3.0.0_ohs_linux64.bin

The Inventory setup screen appears. Enter a valid inventory directory and press OK

Screenshot_8

Press Next in the Welcome screen to begin the installation

Screenshot_9

Select the Oracle Home which we created during the Infrastructure installation

Screenshot_10

Select Collocated HTTP PServer in the next screen. We want to be able to manage the OHS server through weblogic

Screenshot_11

Some checks are done. Make sure all are successful and press Next.

Screenshot_12

A summary is shown. Press Install to begin the installation

Screenshot_13

Watch the installation progress and press Next.

Screenshot_14

Press Finish to end the installation.

Screenshot_15

RCU

The Fusion Middleware Infrastructure needs a few database schema’s to run. These can be crreated with the RCU utility, which is included in the Oracle Home

Run the following command to open RCU

$ORACLE_HOME/oracle_common/bin/rcu

Press Next in the Welcome Screen

Screenshot_17

Select Create Repository > System Load and Product Load

Screenshot_18

Enter the data to connect to your database

Screenshot_19

Select all AS Common Schemas and provide a prefix. This will prepend the schema owners.

Screenshot_20

Prerequisites are checked.

Screenshot_21

Provide a password for the schemas, or specifiy different passwords for each schema.

Screenshot_22

Accept the default tablespaces and press Next.

Screenshot_23

Tablespaces are created. Press OK to continue

Screenshot_24

Press Create in the next screen to actually create the schemas.

Screenshot_25

If all is successful, press Close in the last screen.

Screenshot_26

 

We are now ready to create a domain!

Creating the Domain

 

Screenshot_27

Screenshot_28

Screenshot_29

Screenshot_30

Screenshot_31

Screenshot_32

Screenshot_33

Screenshot_34

Screenshot_35

Screenshot_36

Screenshot_37

Screenshot_38

Screenshot_39

Screenshot_40

Screenshot_41

Screenshot_42

Screenshot_43

Screenshot_44

Screenshot_45

Screenshot_46

Screenshot_47

Screenshot_48

Screenshot_49